Imaging device, endoscope apparatus, and imaging method

ABSTRACT

An imaging device includes an image sensor, an optical system forming an image of an object on the image sensor, and a processor. The optical system switches between a first state of capturing an image of the object with a single pupil and a second state of capturing an image of the object with two pupils. The processor generates a simulative phase difference image from a first captured image captured with the image sensor in the first state, and executes matching processing of comparing the simulative phase difference image with a second capture image captured with the image sensor in the second state to detect a phase difference between an image formed with one of the two pupils and an image formed with another one of the two pupils.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/JP2015/083774, having an international filing date of Dec. 1,2015, which designated the United States, the entirety of which isincorporated herein by reference.

BACKGROUND

The present invention relates to an imaging device, an endoscopeapparatus, an imaging method, and the like.

Techniques for optically measuring a three-dimensional shape haveconventionally been known, with various methods for the measuringproposed. The proposed methods include: stereoscopic imaging based on astereoscopic view with both left and right eyes; phase shift bypatterned illumination using a sinusoidal pattern and the like; and Timeof Flight (TOF) based on time measurement for reflected light.

The stereoscopic imaging can be achieved with a simple mechanism with astereoscopic optical system used for an imaging system, and thusrequires no special illumination mechanisms or illumination control, andalso requires no advanced signal processing. Thus, this technique can besuitably implemented in a small space and thus is advantageous in animaging system that has been progressively downsized recently. Forexample, the technique can be applied to an end of an endoscopeapparatus, to a visual sensor in a small robot, and for various otherneeds. Such an application is likely to require not only a highlyaccurate measurement function but also a normal observation functionwith high image quality. Thus, to ensure a sufficient resolution, it isa common practice to form parallax images on a common image sensorinstead of using separate image sensors. The basic idea of thestereoscopic imaging is to obtain a distance to an object based on anamount of parallax between left and right images. If the left and rightimages fail to be separately formed on the common image sensor, theamount of parallax cannot be detected, and thus the distance informationcannot be obtained.

JP-A-2014-28008 discloses an example of a method of separately formingleft and right images. Specifically, switching between left and rightimaging optical paths is performed along time with a mechanical shutter,so that the left and the right images are obtained. In this method,white light may be used for illumination, for example.

The left and the right images, separately obtained by the methodaccording to JP-A-2014-28008 in a time division manner, can each be usedas an observation image.

SUMMARY

According to one aspect of the invention, there is provided an imagingdevice comprising: an image sensor;

an optical system forming an image of an object on the image sensor; and

a processor,

the optical system switching between a first state of capturing an imageof the object with a single pupil and a second state of capturing animage of the object with two pupils,

the processor being configured to implement generating a simulativephase difference image from a first captured image captured with theimage sensor in the first state, and executing matching processing ofcomparing the simulative phase difference image with a second captureimage captured with the image sensor in the second state to detect aphase difference between an image formed with one of the two pupils andan image formed with another one of the two pupils.

According to another aspect of the invention, there is provided anendoscope apparatus comprising the above imaging device.

According to another aspect of the invention, there is provided animaging method comprising:

switching a state of an optical system between a first state in whichthe optical system forms an image of an object on an image sensor withone pupil and a second state in which the optical system forms the imageof the object on the image sensor with two pupils,

generating a simulative phase difference image from a first capturedimage captured with the image sensor in the first state,

executing matching processing to compare the simulative phase differenceimage with a second captured image captured with the image sensor in thesecond state, and

detecting a phase difference between an image formed with one of the twopupils and an image formed with another one of the two pupils.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a basic configuration of an imagingsection of an endoscope apparatus.

FIG. 2 further illustrates the example of the basic configuration of theimaging section of the endoscope apparatus.

FIG. 3 illustrates an example of a detailed configuration of a fixedmask and a movable mask.

FIG. 4 further illustrates the example of the detailed configuration ofthe fixed mask and the movable mask.

FIG. 5 illustrates an example of a configuration of the endoscopeapparatus.

FIG. 6 illustrates a phase difference detection method.

FIG. 7 illustrates phase difference detection method taking a motioninto consideration.

FIG. 8 illustrates a principle of stereoscopic three dimensionalmeasurement.

FIG. 9 illustrates a detailed configuration example the endoscopeapparatus.

FIG. 10 illustrates a first sequence of operations in movie capturing.

FIG. 11 illustrates a second sequence of operations in movie capturing.

FIG. 12 illustrates a second configuration example of the imagingsection of the endoscope apparatus.

FIG. 13 illustrates the second configuration example of the imagingsection of the endoscope apparatus.

FIG. 14 illustrates a second detail configuration example of the fixedmask and the movable mask.

FIG. 15 illustrates the second detail configuration example of the fixedmask and the movable mask.

FIG. 16 illustrates the third detail configuration example of the fixedmask and the movable mask.

FIG. 17 illustrates the third detail configuration example of the fixedmask and the movable mask.

FIG. 18 illustrates a third phase difference detection method.

FIG. 19 illustrates the third phase difference detection method.

FIG. 20 illustrates the third phase difference detection method.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Some aspects of the present invention can provide an imaging device, anendoscope apparatus, an imaging method with which a capturing of highresolution image of stereoscopic measurement can both be achieved.

According to one embodiment of the invention, there is provided animaging device comprising: an image sensor;

an optical system forming an image of an object on the image sensor; and

a processor,

the optical system switching between a first state of capturing an imageof the object with a single pupil and a second state of capturing animage of the object with two pupils,

the processor being configured to implement generating a simulativephase difference image from a first captured image captured with theimage sensor in the first state, and executing matching processing ofcomparing the simulative phase difference image with a second captureimage captured with the image sensor in the second state to detect aphase difference between an image formed with one of the two pupils andan image formed with another one of the two pupils.

According to one aspect of the present embodiment, the simulative phasedifference image is generated from the first captured image obtainedwith a single pupil and the matching processing is executed to comparethe simulative phase difference image with the second captured imagecaptured with two pupils to detect the phase difference. Thus, thestereoscopic measurement can be performed with a phase differencedetected from the second captured image obtained by superimposing imagesobtained with two pupils. With this configuration, images obtained withdifferent pupils need not to be formed on different areas of the imagesensor, and can be formed in a large area of the image sensor in asuperimposed manner, whereby a high resolution image can be captured.

The present embodiment will be described below. The present embodimentdescribed below does not unduly limit the scope of the present inventiondescribed in the appended claims. Not all the components described inthe present embodiment are required to embody the present invention.

In the description below, an example where the present invention isapplied to an industrial endoscope apparatus is described. However, theapplication of the present invention is not limited to industrialendoscope apparatuses. The present invention may be applied to anythree-dimensional measurement device that measures a three-dimensionalshape through stereoscopic imaging (a method of acquiring distanceinformation on an object by detecting a phase difference between twoimages obtained with an imaging system involving parallax), and to anyimaging device having a three-dimensional measurement function (such asa medical endoscope apparatus, a microscope, an industrial camera, and avisual function of a robot, for example).

1. Basic Configuration

For example, an examination using an endoscope apparatus is performed asfollows. A scope is inserted into an examination target to check whetherthere is an abnormality while capturing normal images. When a portion,such as a defect, to be observed in detail is found, thethree-dimensional shape of the portion is measured to determine whethera further examination is required. Thus, the normal observation image iscaptured with white light. For example, stereoscopic imaging may beperformed with white light so that stereoscopic measurement and theimage capturing with white light can both be achieved. The stereoscopicimaging using white light requires an image sensor to be divided intoleft and right regions, and a left image and a right image to berespectively formed on the left and the right regions. Thus, only animage with a low resolution can be obtained. A color-phase differencemethod may be employed to form the left and the right images on a singleregion of the image sensor. Unfortunately, this method results in acaptured image with color misregistration that is unacceptable as theobservation image.

In view of the above, time-division switching (for example,JP-A-2014-28008) is required for forming the left and the right imageson the single region of the image sensor with white light. However,relative movement between an imaging system and an object leads toshifting due to the movement between the left and the right images,resulting in inaccurate triangulation. Devices such as endoscope cannothave a camera fixed relative to the object and thus are highly likely toinvolve motion blur.

With the present embodiment, an observation image with high resolutioncan be captured with white light, and the stereoscopic measurement in anon-time-division manner, not based on the color-phase differencemethod, can be performed using captured images based on light that haspassed through the left and the right pupils to be incident on the samearea of the image sensor. Thus, the problems described above can besolved. The present embodiment enables stereoscopic measurement and acapturing of an observation image to be performed in real time.

An application of the present invention described below includes adevice having an imaging system that is not stably positioned (fixed)and having an imaging mechanism too small to use a large image sensorfor ensuring a sufficient resolution. A typical example of such a deviceincludes an industrial endoscope. Still the application of the presentinvention is not limited to such a device, and the present invention canbe widely applied to a three-dimensional measurement device directed tohigh-resolution monitoring and highly accurate measurement.

FIG. 1 and FIG. 2 illustrate a basic configuration example of an imagingsection of an endoscope apparatus. FIG. 1 and FIG. 2 each include across-sectional view of an imaging section as viewed in a lateraldirection (on a plane including an optical axis) and a graphillustrating a relationship between an amount of light of an imageformed on the image sensor (or a pixel value of an image formed on theimage sensor) and a position x. The position x is a position(coordinate) in a direction orthogonal to the optical axis of theimaging optical system, and is a pixel position of the image sensor forexample. Although the position is actually defined in a two-dimensionalcoordinate system, the position is described based on a one-dimensionalcoordinate system corresponding to a parallax direction in the twodimensional coordinate system. An illumination mechanism is omitted inthe figures used in the description below.

The endoscope apparatus according to the present embodiment includes anoptical system 15 (optical device) and an image sensor 40. The opticalsystem 15 includes an imaging optical system 10, a movable mask 30(first mask), and a fixed mask 20 (second mask).

The imaging optical system 10 includes a left-eye imaging system 11(first imaging optical system) and a right-eye imaging system 12 (secondimaging optical system) forming a stereoscopic optical system. Forexample, each of the left- and the right-eye imaging optical systemsincludes one or a plurality of lenses, and forms an image of an objectentirely over (or on a major part of) the pixel array of the imagesensor 40. For example, image circles of the left- and the right-eyeimaging optical systems largely overlap. The pixel array of the imagesensor is placed within this overlapping area. In the figure, drepresents a distance between an optical axis AX1 of the left-eyeimaging system 11 and an optical axis AX2 of the right-eye imagingsystem 12, serving as a baseline length in the stereoscopic measurement.

For example, the image sensor 40 includes a color filter with RGBBayerpattern. However, this should not be construed in a limiting sense.For example, a complementary color filter or the like may be provided.

For example, the fixed mask 20 and the movable mask 30 are disposed at apupil position of the imaging optical system 10. The fixed mask 20 isfixed with respect to the imaging optical system 10, whereas the movablemask 30 can have the position switched on a plane orthogonal to theoptical axes AX1 and AX2. Thus, the movable mask 30 can achieve highspeed switching between a first state illustrated in FIG. 1,corresponding to the observation mode (monitoring mode, first mode,first state), and in a second state, corresponding to a stereoscopicmeasurement mode (stereoscopic measurement mode, second state, secondmode) illustrated in FIG. 2. The movable mask 30 is a light shieldingsection (light shielding member). The size of the movable mask 30 is setin such a manner that one of two stop holes of the fixed mask 20 can becovered with the light shielding section in the first mode. FIG. 1 andFIG. 2 each illustrate a configuration where the movable mask 30 isdisposed more on the image side than the fixed mask 20. Alternatively,the movable mask 30 may be disposed more on the object side than thefixed mask 20.

One of two optical paths, that is, one of left- and right-eye opticalpaths or both is selected as an imaging optical path of the imagingoptical system 10 with the fixed mask 20 and the movable mask 30.

FIG. 1 illustrates a state (observation mode) for obtained a normalobservation image. In this state, the right-eye optical path is blocked(shielded) with the movable mask 30 and only the left-eye optical pathcorresponding to the stop hole of the fixed mask 20 is open. Thus, animage IL formed on the image sensor 40 is obtained with the left-eyeimaging system 11 only, whereby a normal captured image (obtained with asingle optical system and white light) is obtained.

FIG. 2 illustrates a state (stereoscopic measurement mode) forsimultaneously obtaining left and right stereoscopic images. In thisstate, the left and the right optical paths are open with the movablemask 30, and an image (double image) as a result of superimposing a leftpupil image IL′ with a right pupil image IR′ is obtained. The left pupilimage IL′ and the right pupil image IR′ are each a white light image.The white light image is an image captured based on spectralcharacteristics of a color filter of an image sensor, and includes a redcolor component, a green color component, and a blue color component.The color filter may have an infrared range. In such a case, thecomponents of the colors of the image may each include an infraredcomponent.

2. Fixed Mask and Movable Mask

FIG. 3 and FIG. 4 illustrate a detailed configuration example of thefixed mask 20 and the movable mask 30. FIG. 3 and FIG. 4 each include across-sectional view of the imaging optical system 10, the fixed mask20, and the movable mask 30, and a diagram illustrating the fixed mask20 and the movable mask 30 as viewed in the optical axis direction (aback view as viewed from the image side).

The left-pupil optical path of the fixed mask 20 has a stop hole 21. Theright-pupil optical path has a stop hole 22. The stop holes 21 and 22are formed on a light shielding section 24 (shielding member), and areeach in an open state (through hole). The stop holes 21 and 22 arearranged on the same circle with the rotational shaft 35 at the center,for example. The stop holes 21 and 22 have the centers (the center of acircle for example) respectively matching the optical axes AX1 and AX2.The light shielding section 24 is a plate-shaped member provided to beorthogonal with respect to the optical axes AX1 and AX2 for example, toshield a casing, including the imaging optical system 10, in front view(or back view) of the casing.

The movable mask 30 includes a light shielding section with no stophole. The light shielding section is connected to a rotational shaft 35orthogonal to the optical axes AX1 and AX2, and is a plate-shaped memberprovided to be orthogonal to the optical axes AX1 and AX2 for example.The light shielding section has a form of a bar (with one end connectedto the rotational shaft 35). However, this should not be construed in alimiting sense, and any shape may be employed as long as the statesillustrated in FIG. 3 and FIG. 4 can be achieved.

The movable mask 30 rotates about the rotational shaft 35 by apredetermined angle in the direction orthogonal to the optical axes AX1and AX2. For example, this rotational motion can be implemented with apiezoelectric element, a motor, or the like. In the observation modeillustrated in FIG. 3, the left-eye optical path (stop hole 21) of thefixed mask 20 is in an open state and the right-eye optical path (stophole 22) is in a shielded state, as a result of the rotation of themovable mask 30 toward the right-eye side by the predetermined angle. Inthe stereoscopic measurement mode illustrated in FIG. 4, the movablemask 30 is returned to a state with a rotational angle of 0 degrees. Asa result, the left- and the right-pupil optical paths (stop holes 21 and22) of the fixed mask 20 are in the open state.

The stop holes 21 and 22 are holes with sizes corresponding to the depthof field required for capturing an observation image (for example,circular holes with a size defined with a diameter). FIG. 1 to FIG. 4illustrate a case where an area φL of the stop hole 21 and an area φR ofthe stop hole 22 are different from each other. Note that the area φL ofthe stop hole 21 and the area φR of the stop hole 22 may be the same.When the areas are different, the stop hole 22 is smaller than the stophole 21 for example. In FIG. 3 and FIG. 4, φL>φR holds true. However,this should not be construed in a limiting sense, and a configurationsatisfying φL<φR may be employed.

In the description above, the two states are established with themovable mask 30 rotated by the predetermined angle about the shaft.However, this should not be construed in a limiting sense. For example,the two states may be established with a sliding motion of the movablemask 30. For example, the rotational motion or the sliding motion can beimplemented with a magnet mechanism, a piezoelectric mechanism, or thelike that may be appropriately selected to achieve a high speed motionand high resistance.

3. Endoscope Apparatus

FIG. 5 illustrates a configuration example of an endoscope apparatus (animaging device in a broad sense) according to the present embodiment.The endoscope apparatus includes a processing section 210 (processingdevice, processor), an imaging section 105, a storage section 410(memory), an operation section 420 (operation device), and a displaysection 220 (display device, display). The processing section 210includes a phase difference detection section 330 and an image outputsection 325.

The processing section 210 controls the sections of the endoscopeapparatus and executes various types of information processing includingimage processing for example. The processing section 210 includes thephase difference detection section 330, the image output section 325,and a mode control section 345. For example, the storage section 410stores therein image data on an image captured with the imaging section105, setting data on the endoscope apparatus, and the like.Alternatively, the storage section 410 is used as a temporally storagememory (working memory) for the processing section 210. The imagingsection 105 captures an image (movie or a still image). The imagingsection 105 includes the image sensor 40 and the optical system 15. Theimaging section 105 may further include a driving device that drives afocus mechanism of the optical system 15. The operation section 420 isan input device enabling the user to operate the imaging device, and mayinclude a button, a lever, a rotation ring, a mouse, a keyboard, a touchpanel, and the like. The display section 220 displays an image that hasbeen captured with the imaging section 105 and an image that has beenprocessed by the processing section 210. Examples of a display section400 include a liquid crystal display device, an electro-luminescence(EL) display device, and the like.

A configuration and an operation of the endoscope apparatus according tothe present embodiment are described in detail below.

The optical system 15 switches between the first state and the secondstate. In the first state, an image of an object 5 is formed with asingle pupil. In the second state, an image of the object is formed withtwo pupils. The phase difference detection section 330 generates asimulative phase difference image from a first captured image IL(x)captured with the image sensor 40 in the first state. The phasedifference detection section 330 executes matching processing to comparethe simulative phase difference image with a second captured imageILR′(x) captured with the image sensor in the second state. Then, thephase difference detection section 330 detects a phase differencebetween an image formed with one of the two pupils and an image formedwith the other one of the pupils.

The first state corresponds to an observation image acquisition state(observation mode) in FIG. 1 and FIG. 3. The second state corresponds toa measurement image acquisition state (stereoscopic measurement mode) inFIG. 2 and FIG. 4. A single pupil in the first state corresponds to thestop hole 21 of the fixed mask 20. The two pupils in the second statecorrespond to the stop holes 21 and 22 in the fixed mask 20. In FIG. 1to FIG. 4, the two pupils are illustrated as left and right pupils. Notethat the direction in which the two pupils are separated from each otheris not limited to the left and right direction.

The simulative phase difference image (combined image, simulated image,simulated phase difference image) is an image, simulating the secondcaptured image ILR′(x), obtained by combining the first captured imageIL(x) with an image as a result of providing a simulative phasedifference to the first captured image IL(x) to shift a position.Providing the simulative (intentional) phase difference corresponds toproviding a variable corresponding to an appropriate phase differenceand shifting a position x by an amount corresponding to the variable.This corresponds to converting the first captured image IL(x) into animage IL(x−s). In the matching processing, a phase difference with whichthe images match is searched for while changing the simulative phasedifference to be provided. A detail description on such a method ofdetecting the phase difference will be given with reference to FIG. 6,FIG. 7, and FIG. 18 to FIG. 20. In the detailed description, thesimulative phase difference image corresponds to an image ILR(x,s) inFormula (2) described later, an image ILR(x,δ,s) in Formula (4)described later, and a vector NCV in Formula (26) described later. Thesimulative phase difference to be provided corresponds to s and thephase difference detected corresponds to s′(xL).

In the present embodiment, the simulative phase difference image isgenerated from the first captured image IL(x) obtained with a singlepupil. The matching processing is executed to compare the simulativephase difference image with the second captured image ILR′(x) obtainedwith two pupils, so that the phase difference is detected. The secondcaptured image ILR′(x) is obtained by superimposing images, obtainedwith two pupils at once, with each other. This means that the images arenot obtained with the pupils in a time division manner, whereby thephase difference is not affected by movement of the object or theimaging system. Thus, an accurate phase difference (an object distance)unaffected by the movement of the imaging system can be detected. Thematching processing is executed with the simulative phase differenceimage, simulating the second captured image ILR′(x), generated from thefirst captured image IL(x). Thus, the phase difference can be detectedfrom the second captured image ILR′(x) obtained by superimposing theimages, obtained with the two pupils, with each other. With the phasedifference being capable of being detected from the superimposed image,a color-phase difference method needs not to be employed, whereby thestereoscopic measurement can be implemented with white light. The firstcaptured image IL(x) captured with a single pupil can be provided as anobservation image. Thus, the stereoscopic measurement and provision ofthe observation image can both be achieved.

The endoscope apparatus (imaging device) according to the presentembodiment may have the configuration described below. Specifically, theendoscope apparatus according to the present embodiment includes theimage sensor 40, the optical system 15, a memory (storage section 410)that stores information (for example, a program and various types ofdata), and a processor (a processing section 210, a processor includinghardware) that operates based on the information stored in the memory.The processor executes phase difference detection processing including:generating the simulative phase difference image from the first capturedimage IL(x); executing the matching processing to compare the simulativephase difference image with the second captured image ILR′(x); anddetecting a phase difference between an image formed with one of the twopupils and an image formed with the other one of the two pupils.

For example, the function of each section may be implemented by theprocessor or may be implemented by integrated hardware. For example, theprocessor may include hardware, and the hardware may include at leastone of a circuit that processes a digital signal and a circuit thatprocesses an analog signal. For example, the processor may include oneor more circuit devices (e.g., IC), and one or more circuit elements(e.g., resistor or capacitor) that are mounted on a circuit board. Theprocessor may be a central processing unit (CPU), for example. Note thatthe processor is not limited to a CPU. Various other processors such asa graphics processing unit (GPU) or a digital signal processor (DSP) mayalso be used. The processor may be a hardware circuit that includes anASIC. The processor may include an amplifier circuit, a filter circuit,and the like that process an analog signal. The memory may be asemiconductor memory (e.g., SRAM or DRAM), or may be a register. Thememory may be a magnetic storage device such as a hard disk drive (HDD),or may be an optical storage device such as an optical disc device. Forexample, the memory stores a computer-readable instruction, and thefunction of each section of the processing section 210 is implemented bycausing the processor to perform the instruction. The sections of theprocessing section 210 includes a phase difference detection section330, an image output section 325, a mode control section 345, and anerror detection section 355 in FIG. 5. The sections of the processingsection 210 further includes an image selection section 310, a colorimage generating section 320, the phase difference detection section330, a movable mask control section 340, a movable mask positiondetection section 350, a distance information calculation section 360,and a three-dimensional information generating section 370 in FIG. 9.The instruction may be an instruction set that is included in a program,or may be an instruction that instructs the hardware circuit included inthe processor to operate.

For example, operations according to the present embodiment areimplemented as follows. The processor performs control to switch theoptical system 15 between the first state and the second state. Thefirst captured image IL(x) and the second captured image ILR′(x)captured with the image sensor 40 are stored in the memory (storagesection 410). The processor reads the first captured image IL(x) fromthe memory to generate a simulative phase difference image, and storesthe simulative phase difference image in the memory. The processor readsthe second captured image ILR′(x) and the simulative phase differenceimage from the memory, executes the matching processing to compare theimages to detect the phase difference, and stores the phase differencethus detected in the memory.

The sections of the processing section 210 according to the presentembodiment are implemented as modules of a program operating on theprocessor. For example, the phase difference detection section 330 isimplemented as a phase difference detection module that generates thesimulative phase difference image from the first captured image IL(x),and executes the matching processing to compare the second capturedimage ILR′(x) with the simulative phase difference image to detect thephase difference between an image formed with one of the two pupils andan image formed with the other one of the two pupils.

In the present embodiment, the phase difference detection section 330generates a first simulative pupil image and a second simulative pupilimage, respectively formed with one and the other one of the pupils,from the first captured image IL(x). The phase difference detectionsection 330 generates the simulative phase difference image throughprocessing of adding together the first simulative pupil image and thesecond simulative pupil image shifted from each other by a shiftedamount corresponding to the phase difference. The phase differencedetection section 330 detects the phase difference by executing thematching processing while changing the shifting amount.

The first simulative pupil image (first pupil image, first simulatedpupil image) is an image simulating an image formed with one of thepupils, as a part of the second captured image ILR′(x), based on thefirst captured image IL(x). The second simulative pupil image (secondpupil image, second simulated pupil image) is an image simulating animage formed with the other one of the pupils, as a part of the secondcaptured image ILR′(x), based on the first captured image IL(x).Simulating the pupil image corresponds to providing the variablecorresponding to an appropriate phase difference and shifting relativepositions of the first simulative pupil image and the second simulativepupil image by an amount corresponding to the variable. In a methoddescribed later with reference to FIG. 6, the first and the secondsimulative pupil images respectively correspond to the image IL(x) andan image IL(x,s) in Formula (1) described later. The simulative phasedifference image ILR(x,s) is obtained through processing of adding thefirst and the second simulative pupil images together as in Formula (2)described later. In a method described later with reference to FIG. 7,the first and the second simulative pupil images respectively correspondto images IL(x,δ) and IL(x,δ,s) in Formula (3) described later. Thesimulative phase difference image ILR(x,δ,s) is obtained throughprocessing of adding the first and the second simulative pupil images asin Formula (4) described later. In a method described later withreference to FIG. 18 to FIG. 20, the first and the second simulativepupil images respectively correspond to vectors RL and RR in Formula(21) described later. The simulative phase difference image (vector NCV)is obtained through processing of adding the first and the secondsimulative pupil images (vectors RL and RR) as in Formula (26) describedlater.

In the present embodiment, the images formed with the two pupils aregenerated from the first captured image IL(x), and the simulative phasedifference image is generated through the processing of adding theimages together. Thus, the image simulating the second captured imageILR′(x) based on a certain assumed phase difference (shifting amount) isgenerated. Then, through the matching processing executed while changingthe assumed phase difference, the phase difference for the secondcaptured image ILR′(x), obtained by superimposing the images, obtainedwith the two pupils, with each other, can be searched for anddetermined.

In the present embodiment, the two pupils of the optical system 15 havedifferent sizes.

The size of the pupil corresponds to an area of an opening of the pupil.For example, the size of the pupil may be directly represented by thearea of the opening, or may be represented by a parameter of a shape ofthe opening that can be used instead of the area. For example, when theopening has a shape of a circle, the size of the pupil may berepresented by a diameter of the circle.

The matching processing is based on local area comparison that mayresult in a high pseudo correlation for similar images (waveforms). Inview of this, the present embodiment features the two pupils ofdifferent sizes. Thus, captured images obtained with the two pupils aredifferent from each other in brightness, whereby the second capturedimage ILR′(x) can be characterized. This ensures a lower risk of thematching processing resulting in pseudo correlation, whereby moreaccurate phase difference detection can be achieved.

As described later with reference to FIG. 12 to FIG. 15, the imagingoptical system 10 of the optical system 15 may be a monocular opticalsystem. In such a configuration, whether the phase difference is of apositive value or a negative value is determined based on a focusposition (whether the focus position is shifted forward or rearward) asin Formula (17) described later. With the configuration where thecaptured images obtained with the pupils have the same brightness, thesecond captured image ILR′(x) is the same regardless of whether thephase difference is of a positive or a negative value if the absolutevalue of the phase difference is the same. Thus, the phase differencecannot be detected. In view of this, with the present embodiment, thecaptured images obtained with the pupils are different from each otherin brightness, so that the phase difference can be accurately detectedwith the monocular imaging optical system 10.

When the imaging optical system 10 is a stereoscopic optical system asillustrated in FIG. 1 to FIG. 4, the optical system 15 may have twopupils of the same size.

In the present embodiment, the phase difference detection section 330executes gain adjustment, based on different sizes of the two pupils, onthe first captured image IL(x) to generate the first simulative pupilimage corresponding to an image formed with one of the pupils and thesecond simulative pupil image corresponding to an image formed with theother one of the pupils. The phase difference detection section 330generates the simulative phase difference image through the processingof adding together the first simulative pupil image and the secondsimulative pupil image shifted from each other by an amountcorresponding to the phase difference. Then, the phase differencedetection section 330 detects the phase difference by executing thematching processing while changing the shifting amount.

In the method described later with reference to FIG. 6, the gainadjustment corresponds to processing of multiplying the image IL(x−s) bya coefficient (φR/φL) as in Formula (1) described later. In the methoddescribed later with reference to FIG. 7, the gain adjustmentcorresponds to processing of multiplying the image IL(x−δ−s) by thecoefficient (φR/φL) as in Formula (3) described later. The coefficient(φR/φL) is an area ratio between the openings of the pupils. In themethod described later with reference to FIG. 18 to FIG. 20, the gainadjustment corresponds to processing of multiplying vectors VL and VR bycoefficients gL and gR as in Formula (26) described later. Thecoefficients gL and gR represent a ratio between sizes of the vectors(brightness of the images) as can be seen in Formula (25) describedlater. The ratio is related to the area ratio between the openings ofthe pupils (the ratio may be substantially the same as the area ratio,but is not necessarily the same as the area ratio).

In the present embodiment, the simulative phase difference image can beappropriately generated with the two pupils of the optical system 15with difference sizes. Specifically, the gain adjustment based on thesizes of the openings is executed on the first captured image IL(x) thatis captured with a single pupil, so that the simulative phase differenceimage simulating the second captured image ILR′(x) captured with twopupils with openings with different sizes can be generated.

In the present embodiment, the phase difference detection section 330detects a motion amount due to an object moving between the firstcaptured image IL(x) and the second captured image ILR′(x), based on thefirst captured image IL(x) and the second captured image ILR′(x).

The object movement is a movement (shifting) of an imaging position ofthe object between two images captured at different timings. The objectmovement is caused by one of a movement (shifting) of the object, amovement (shifting) of the imaging system, or both.

The first captured image IL(x) and the second captured image ILR′(x) arecaptured in a time division manner, and thus the phase difference mightnot be accurately detectable when the object movement occurs between theimage capturing timings. In the present embodiment, the phase differencecan be detected without being affected by the object movement, with themotion amount between the first captured image IL(x) and the secondcaptured image ILR′(x) further detected. The images are simultaneouslyobtained with the two pupils to be the second captured image ILR′(x)including information on the phase difference not affected by themovement. Thus, the object movement and the phase difference can beseparately detected.

In the present embodiment, the phase difference detection section 330generates the simulative phase difference image through the processingof adding together the first simulative pupil image and the secondsimulative pupil image, shifted from each other by a first shiftingamount corresponding to the phase difference and by a second shiftingamount corresponding to the motion amount. The phase differencedetection section 330 detects the phase difference and the motion amountby executing the matching processing with the first shifting amount andthe second shifting amount changed independently from each other.

In the methods described later with reference to FIG. 7 and FIG. 18 toFIG. 20, s and δ respectively correspond to the first shifting amount(phased difference) and the second shifting amount (motion amount), forgenerating the simulative phase difference image. In the methods, s′(xL)and δ′(xL) respectively represent the phase difference detected by thematching processing and the motion amount.

In the present embodiment, the images obtained with the two pupils aregenerated from the first captured image IL(x), and the simulative phasedifference image is generated through the processing of adding theimages together. Thus, an image simulating the second captured imageILR′(x) based on a certain assumed phase difference (first shiftingamount) and a certain assumed motion amount (second shifting amount).The information on the phase difference in the second captured imageILR′(x) can be extracted and the motion amount between the firstcaptured image IL(x) and the second captured image ILR′(x) can bedetected by executing the matching processing by changing the assumedphase difference and the assumed motion amount.

In the present embodiment, the optical system 15 is set to be in thefirst state in an n-th frame (n is an integer) and is set to be in thesecond state in an n+1-th frame to an n+j-th frame (j is an integerequal to or larger than 2) after the n-th frame. The phase differencedetection section 330 detects the phase difference for j times based onthe first captured image IL(x) captured in the n-th frame and the secondcaptured image ILR′(x) captured in an n+i-th frame (i is an integer thatis equal to or larger than 1 and is equal to or smaller than j), andexecutes processing of averaging the j phase differences.

This processing is described in detail later with reference to FIG. 11.In FIG. 11, a final phase difference s′(xL) is obtained by theprocessing of averaging the j phase differences with Formula (11)described later where j is 5.

Considering the object movement, logically, the time interval betweenthe image capturing in the first state and the image capturing in thesecond state should be short as much as possible. However, in thepresent embodiment, the motion amount and the phase difference can bedetected independently from each other. This allows a certain length oftime interval to be set between the image capturing in the first stateand the image capturing in the second state without compromising theaccuracy of the phase difference detection. Thus, the final phasedifference can be obtained by averaging a plurality of phase differencesobtained by sequentially capturing images for a plurality of times inthe second state after a single image has been captured in the firststate. The processing of averaging the plurality of phase differencesensures more accurate phase difference detection.

In the present embodiment, the endoscope apparatus includes the imageoutput section 325 that outputs an observation image based on the firstcaptured image IL(x).

In the present embodiment, the stereoscopic measurement based on thefirst captured image IL(x) and the second captured image ILR′(x) and theoutput of the observation image based on the first captured image IL(x)can both be achieved. This ensures capturing of the observation imageand the stereoscopic measurement for an object in the image to besubstantially simultaneously implemented. As will be described laterwith reference to FIG. 10 and FIG. 11, this processing may be applied toa case where a movie is captured so that capturing of the observationimage and stereoscopic measurement can be implemented substantially inreal-time.

In the present embodiment, the optical system 15 includes the fixed mask20 including first and second openings and the movable mask 30 that ismovable relative to the fixed mask 20. In the first state, the opticalsystem 15 forms an image of the object 5 with the first opening servingas the single pupil, with the movable mask 30 not shielding the firstopening and shielding the second opening. In the second state, theoptical system 15 forms the image of the object 5 with the first openingand the second opening serving as the two pupils, with the movable mask30 shielding none of the first opening and the second opening.

In FIG. 3 and FIG. 4, the first opening corresponds to the stop hole 21and the second opening corresponds to the stop hole 22. The mask is amember or a component shielding light incident on the mask. The fixedmask 20 according to the present embodiment has a light shieldingsection 24 shielding light and the stop holes 21 and 22 transmittinglight. The movable mask 30 is formed of a light shielding section withno opening, and shields light.

In the present embodiment, the first state of forming the image of theobject with a single pupil can be established with the movable mask 30shielding the second opening without shielding the first opening. Thesecond state of forming the image of the object with two pupils can beestablished with the movable mask 30 not shielding the first and thesecond openings.

In the present embodiment, the second opening is smaller than the firstopening on the fixed mask 20.

In the present embodiment, two pupils with different sizes can beimplemented. For example, the size of an opening is represented by aparameter of the shape of the opening that can be used instead of thearea. For example, when the opening has a shape of a circle, the size isrepresented by a diameter of the circle. Alternatively, the area of theopening may be directly used as the size of the opening.

In the present embodiment, the fixed mask 20 and the movable mask 30 maybe configured as follows. Specifically, the fixed mask includes anopening. In the first state, the movable mask 30 does not split anopening (sets the opening to be in an open state), and the opticalsystem 15 forms an image of the object 5 with the opening that is notsplit serving as the single pupil. In the second state, the movable mask30 splits the opening into a first split opening and a second splitopening smaller than the first split opening. The optical system 15forms an image of the object 5 with the first and the second splitopenings servings as the two pupils.

This configuration is described in detail with reference to FIG. 16 andFIG. 17. In FIG. 16 and FIG. 17, the opening of the fixed mask 20corresponds to a stop hole 23 and the first and the second splitopenings respectively correspond to holes FL and FR.

In the present embodiment, the first state for forming an image of anobject with a single pupil can be established with the movable mask 30not splitting the opening of the fixed mask 20. The second state offorming an image of an object with two pupils can be established withthe movable mask 30 splitting the opening of the fixed mask 20 into thefirst split opening and the second split opening.

In the present embodiment, the endoscope apparatus includes the modecontrol section 345 that performs control to switch between a first mode(observation mode) of setting the optical system 15 to be in the firststate and a second mode (stereoscopic measurement mode) of setting theoptical system 15 to be in the second state.

The mode control section 345 is described in detail with reference toFIG. 9. The movable mask control section 340 in FIG. 9 corresponds tothe mode control section 345.

In the present embodiment, the state of the optical system 15 can beswitched between the first state and the second state, through modesetting by the mode control section 345. The phase difference detectionsection 330 can determine whether the captured image is the firstcaptured image or is the second captured image based on mode informationfrom the mode control section 345.

In the present embodiment, the error detection section 355 is providedto detect at least one of the optical system 15 set to be in the firststate under the first mode and the optical system 15 set to be in thesecond state under the second mode, based on the image IL(x) capturedunder the first mode and the image ILR′(x) captured under the secondmode.

The error detection section 355 is described in detail with reference toFIG. 9. The movable mask position detection section 350 in FIG. 9corresponds to the error detection section 355.

The optical system 15 is configured to be switched between the first andthe second states, and includes a movable section (movable mask 30).When such a movable section is used, a possibility of an erroneousoperation of the movable section needs to be taken into consideration.In the present embodiment, whether or not appropriate switching betweenthe first and the second states is achieved can be detected from theimage. When an error is detected, the image capturing may be stopped orthe operation of the movable section may be corrected, for example. Forexample, it is determined that the normal state has been restored whenthe error is no longer detected as a result of temporally stopping andthen resuming the operation of the movable section.

4. Method of Detecting Phase Difference

A phase difference s between the left pupil image IL′ and the rightpupil image IR′ needs to be detected to obtain the distance to theobject. The image obtained in the stereoscopic measurement modedescribed above is formed by superimposing the left pupil image IL′ andthe right pupil image IR′. Thus, the phase difference s cannot bedetected by using this image only. Thus, the phase difference s isobtained by using the image IL obtained under the observation mode. Thismethod is described below.

FIG. 6 illustrates a method of detecting the phase difference. Thesimple description is given by focusing only on an x coordinate andignoring a y coordinate. Pixel values of the images IL, IL′, and IR′ areregarded as functions of the x coordinate, and will be referred to asIL(x), IL′(x), and IR′(x), respectively. In the actual stereoscopicmeasurement mode, a captured image ILR′(x)=[IL′(x)+IR′(x)] is obtained.Still, IL′(x) and IR′(x) are each illustrated as an individual waveformin FIG. 6. In processing of detecting a phase difference, the imagesIL(x) and ILR′(x) are converted into monochrome images (greyscaleimages), and the phase difference is detected from the monochromeimages, for example.

First of all, the image IL(x−s) is generated by shifting the left pupilimage IL(x) for observation by s from a certain coordinate xL. The stopholes 21 and 22 have different sizes. Thus, gain adjustment based onIL(x) is executed on IL(x−s) as in the following Formula (1) using theratio between the areas φL and φR of the stop holes 21 and 22. An imageafter the gain adjustment is referred to as IL(x,s).IL(x,s)=(ϕR/ϕL)·IL(x−s)  (1)

The adjustment gain (φR/φL) is for gain matching between the left andthe right pupil images. In Formula (1) described above, the gain is setbased on an area ratio between the stop holes 21 and 22. However, thisshould not be construed in a limiting sense, and the optimum gainadjustment may be implemented based on the optical characteristics of anactual imaging system for example.

Next, IL1(x) and IL(x,s) in Formula (1) described above are combined togenerate the combined image ILR(x,s) as in Formula (2) described belowto be a comparison image for the search. ILR(x,s) is an image obtainedby adding together the two images IL(x) (one of which has had the gainadjustment) shifted from each other by the phase difference s.ILR(x,s)=IL(x)+IL(x,s)=IL(x)+(ϕR/ϕL)·IL(x−s)  (2)

Next, matching valuated is performed while changing the shifting amounts to check that matching between the superimposed imageILR′(x)[=IL′(x)+IR′(x)] captured for the measurement and the combinedimage ILR(x,s) corresponding to each value of the shifting amount s.Then, the shifting amount s providing the highest level of matching isdetected to be the phase difference s′(xL) between the left pupil imageIL′(x) and the right pupil image IR′(x) at the coordinate xL. In FIG. 6,w represents a range in which comparison to check the similarity isperformed in the matching evaluation.

In this method, the images IL(x), IL′(x), and IR′(x) correspond todifferent points of view but are regarded as being substantially insimilar relationship locally. Specifically, in a predetermined section,IL(x) and IL′(x) are regarded as substantially matching, and an imageobtained by shifting IL(x) by s and IR′(x) are also regarded asmatching. Thus, whether or not ILR(x) and ILR′(x) in the combined statematch is checked while changing the search value s. When ILR(x) andILR′(x) match, IL(x) and IL′(x) are determined to match and IL(x−s) andIR′(x) are determined to match, and s at this point is determined as aphase difference s′ to be obtained.

5. Method of Detecting Phase Difference while Taking Movement intoConsideration

The phase difference s′ obtained as described above is obtained under anassumption that images are obtained with no movement between the imagingsystem and the object, that is, under an assumption that the image IL(x)and the image IL′(x) are at the same position in an imaging plane.However, the left pupil image IL(x) for the observation and thesuperimposed image ILR′(x) for the measurement are sequentially capturedin a time division manner Thus, there might be movement between theimaging system and the object during a time interval between the timingsat which the images are captured. In such a case, the amount of themovement needs to be taken into consideration to obtain the phasedifference.

FIG. 7 illustrates a method of detecting the phase difference whiletaking the movement into consideration. The simple description is givenby focusing only on the x coordinate and ignoring the y coordinate. Inthe stereoscopic measurement mode, ILR′(x)=[IL′+IR′(x)] is obtained asthe captured image. In FIG. 7, IL′(x) and IR′(x) are each illustrated asan individual waveform.

The amount δ represents the motion amount δ between the left pupil imageIL(x) and the superimposed image ILR′(x) sequentially obtained. The leftpupil image IL(x) for the observation is separated into the imageIL(x,δ) shifted from the coordinate xL by the motion amount δ and theimage IL(x,δ,s) shifted from the coordinate xL by the sum of the motionamount δ and the phase difference s. After these images are thusgenerated, the gain adjustment is executed as in the following Formula(3) based on IL(x) by using the ratio between the areas φL and φR of thestop holes 21 and 22.

$\begin{matrix}\left. \begin{matrix}{{{IL}\left( {x,\delta} \right)} = {{IL}\left( {x - \delta} \right)}} \\{{{IL}\left( {x,\delta,s} \right)} = {\left( {\phi\;{R/\phi}\; L} \right) \cdot {{IL}\left( {x - \delta - s} \right)}}}\end{matrix} \right\} & (3)\end{matrix}$

The images as a result of the gain adjustment are referred to as IL(x,δ)and IL(x,δ,s). In Formula (3) described above, the area ratio betweenthe stop holes 21 and 22 is used for the gain setting. However, thisshould not be construed in a limiting sense. For example, the optimumgain adjustment may be implemented by using the actual imaging system.

Then, IL(x,δ) and IL(x,δ,s) are combined to intentionally generate thecombined image ILR(x,δ,s) as in Formula (4) described later to be usedas the comparison image for the search.ILR(x,δ,s)=IL(x,δ)+IL(x,δ,s)=IL(x−δ)+(ϕR/ϕL)·IL(x−δ−s)  (4)

Next, the matching evaluation is performed between the superimposedimage ILR′(x) for measurement and the combined image ILR(x,δ,s) based onthe phase difference s and the motion amount δ that are individuallychanged. Then, the phase difference s and the motion amount δ providingthe highest level of matching are detected to be the phase differences′(xL) and the motion amount δ′(xL) of the left pupil image IL′(x) andthe right pupil image IR′(x) at each coordinate xL. In FIG. 7, wrepresents the range in which the comparison to check the similarity isperformed in the matching evaluation.

This method is also performed under an assumption that the images IL(x),IL′(x), and IR′(x) are locally substantially the same. However, theimage IL(x), IL′(x), and IR′(x) are images with parallax, and thus areactually different waveforms. Still, when the shifting amount s and themotion amount δ are relatively small, the local ranges of the images canbe regarded as being in similar relationship.

With this method, the phase difference between the left pupil imageIL′(x) and the right pupil image IR′(x) in the measurement stereoscopicimage obtained by superimposing involving the motion amount δ can bedetected without being affected by the motion amount δ. This is becausethe left pupil image IL′(x) and the right pupil image IR′(x) aresimultaneously captured (as the superimposed image) under thestereoscopic measurement mode, and thus the phase difference s isunaffected by the movement. All things considered, the phase differences can be extracted with the motion amount δ separated from the shiftingamount (that is, s+δ) between the images IL(x) and ILR′(x).

6. Principle of Stereoscopic Three-Dimensional Measurement

The principle of the stereoscopic measurement in the configurationexample illustrated FIG. 1 to FIG. 4 is described.

As illustrated in FIG. 8, the optical paths for the left eye and theright eye are each independently formed. Reflected light from the object5 passes through these optical paths so that the object image is formedon the image sensor plane (light receiving surface). A coordinate systemX, Y, Z in the three-dimensional space is defined as follows.Specifically, an X axis and a Y axis orthogonal to the X axis are setalong the image sensor plane. A Z axis, toward the object, is set to bein a direction that is orthogonal to the image sensor plane, andparallel to the optical axes AX1 and AX2. The Z axis, the X axis, andthe Y axis intersect at the zero point. The Y axis is omitted for thesake of illustration.

Here, the distance between the imaging optical system 11, 12 (imaginglens) and the image sensor plane is defined as b, and the distancebetween the imaging optical system 11, 12 and a certain point Q(x,z) ofthe object 5 is defined as z. The optical axes AX1 and AX2 are arrangedto be at the same distance from the Z axis. This distance is defined asd/2. Thus, the baseline length for the stereoscopic measurement is d. AnX coordinate of a corresponding point, corresponding to the certainpoint Q(x,y) of the object 5, as a part of an image formed on the imagesensor plane with the imaging optical system 11 is XL. An X coordinateof the corresponding point, corresponding to the certain point Q(x,y) ofthe object 5, as a part of the image formed on the image sensor planewith the imaging optical system 12 is XR. The following Formula (5) canbe obtained based on a similarity relation among a plurality of partialright angle triangles formed within a triangle defined by the certainpoint Q(x,z) and the coordinates XL and XR.

$\begin{matrix}{\frac{z}{b} = {\frac{{x + {d/2}}}{{{XL} + {d/2}}} = \frac{{x - {d/2}}}{{{XR} - {d/2}}}}} & (5)\end{matrix}$

The following Formulae (6) and (7) hold true.

$\begin{matrix}\left. \begin{matrix}{{x + {d/2}} > {{0\mspace{14mu}{when}\mspace{14mu}{XL}} + {d/2}} < 0} \\{{x + {d/2}} < {{0\mspace{14mu}{when}\mspace{14mu}{XL}} + {d/2}} > 0}\end{matrix} \right\} & (6) \\\left. \begin{matrix}{{x - {d/2}} > {{0\mspace{14mu}{when}\mspace{14mu}{XR}} - {d/2}} < 0} \\{{x - {d/2}} < {{0\mspace{14mu}{when}\mspace{14mu}{XR}} - {d/2}} > 0}\end{matrix} \right\} & (7)\end{matrix}$

Thus, the absolute value in Formula (5) described above can be normalvalues as in the following Formula (8).

$\begin{matrix}{\frac{z}{b} = {{- \frac{x + {d/2}}{{XL} + {d/2}}} = {- \frac{x - {d/2}}{{XR} - {d/2}}}}} & (8)\end{matrix}$

Formula (8) described above can be solved for x as in the Formula (9).

$\begin{matrix}{x = {{- \frac{d}{2}} \cdot \frac{{XR} + {XL}}{{XR} - {XL} - d}}} & (9)\end{matrix}$

The following Formula (10) for obtaining z can be obtained bysubstituting x in Formula (9) described above into Formula (8) describedabove.

$\begin{matrix}{z = {\frac{d}{\left( {{XR} - {XL} - d} \right)} \cdot b}} & (10)\end{matrix}$

Note that d and b are known setting values, and an unknown value (XR−XL)is detected as the phase difference s through the matching processing(correlation calculation) described above. The object shape can bemeasured by calculating the distance z for each position x. Somedistances z might be unobtainable due to matching failure. Suchdistances z may be obtained by interpolation using the distances zobtained for the surrounding pixels or by other like method, forexample.

7. Detailed Configuration of Endoscope Apparatus

FIG. 9 illustrates a detailed configuration example of an endoscopeapparatus (an imaging device in a broad sense). The endoscope apparatusincludes a scope section 100 (imaging section) and a main body section200 (controller device). The scope section 100 includes the opticalsystem 15, the image sensor 40, and the driving section 50. The opticalsystem 15 includes the imaging optical system 10, the fixed mask 20, andthe movable mask 30. The main body section 200 includes a processingsection 210 (processing circuit, processing device), a display section220 (display device), and the imaging processing section 230 (imagingprocessing circuit). The processing section 210 includes the imageselection section 310 (image frame selection unit), the color imagegenerating section 320 (image output section), the phase differencedetection section 330, the movable mask control section 340 (modecontrol section), the movable mask position detection section 350, thedistance information calculation section 360, and the three-dimensionalinformation generating section 370.

The scope section 100, the color image generating section 320, themovable mask control section 340, and the movable mask positiondetection section 350 respectively correspond to the imaging section105, the image output section 325, the mode control section 345, and theerror detection section 355 in FIG. 5. The storage section 410 and theoperation section 420 in FIG. 5 are omitted in FIG. 9. The scope section100 may further include unillustrated components such as a treatmentinstrument and an illumination device (such as a light source and alens).

The endoscope apparatus may be what is known as a video scope (anendoscope apparatus incorporating an image sensor) for industrial andmedical use. The present invention can be applied to a flexibleendoscope with the scope section 100 that is flexible and to a rigidendoscope with the scope section 100 that is in a form of a stick. Forexample, a flexible endoscope for industrial use includes the main bodysection 200 and the scope section 100 serving as a portable device thatcan be carried around. The flexible endoscope is used for inspection inmanufacturing and maintenance processes for industrial products, in amaintenance process for buildings and pipes, and in other likesituations.

The driving section 50 drives the movable mask 30 based on the controlsignal from the movable mask control section 340, to switch between thefirst state (observation mode) and the second state (stereoscopicmeasurement mode). For example, the driving section 50 includes anactuator including a piezoelectric element and a magnet mechanism.

The imaging processing section 230 executes an imaging process on asignal from the image sensor 40, and outputs a captured image (such as aBayer image, for example). For example, a correlative double samplingprocess, a gain control process, an A/D conversion process, gammacorrection, color correction, noise reduction, and the like areexecuted. For example, the imaging processing section 230 may include adiscrete IC such as an ASIC, or may be incorporated in the image sensor40 (sensor chip) and the processing section 210.

The display section 220 displays an image captured by the scope section100, three-dimensional shape information on the object 5, or the like.For example, the display section 220 includes a liquid crystal display,an Electro-Luminescence (EL) display, and the like.

An operation of the endoscope apparatus is described below. The movablemask control section 340 controls the driving section 50, and thusswitches the position of the movable mask 30. When the movable maskcontrol section 340 sets the movable mask 30 to be in the observationmode, an image of the object 5 is formed on the image sensor 40 withreflected light from the object 5 that has passed through the left-eyeoptical path. The imaging processing section 230 reads out pixel valuesof the image formed on the image sensor 40, performs the A/D conversionor the like, and outputs resultant image data to the image selectionsection 310.

The image selection section 310 detects that the movable mask 30 is inthe state corresponding to the observation mode based on the controlsignal from the movable mask control section 340, and outputs thecaptured image IL(x) to the color image generating section 320 and thephase difference detection section 330. The color image generatingsection 320 performs demosaicing process (process for generating an RGBimage from a Bayer image) and various image processes, and outputs theresultant RGB primary color image to the display section 220. Thedisplay section 220 displays this color image.

When the movable mask control section 340 sets the movable mask 30 to bein the stereoscopic measurement mode, images are simultaneously formedon the image sensor 40 based on the reflected light from the object 5,through the left-pupil optical path and the right-pupil optical path.The imaging processing section 230 reads out pixel values of the imageformed on the image sensor 40, performs the A/D conversion or the like,and outputs resultant image data to the image selection section 310.

The image selection section 310 detects that the movable mask 30 is inthe state corresponding to the stereoscopic measurement mode based onthe control signal from the movable mask control section 340, andoutputs the captured image ILR′(x) to the phase difference detectionsection 330. The phase difference detection section 330 convers theimage IL(x) and the image ILR′(x) into monochrome images, executesmatching processing described above on the images thus obtained by theconversion, and detects the phase difference (phase shift) for eachpixel. The phase difference detection section 330 determines whether thedetected phase difference is reliable, and outputs an error flag foreach pixel determined to have an unreliable phase difference. Variousmatching evaluation methods for obtaining the phase difference betweentwo similar waveforms (the image ILR(x) and the image ILR′(x)) haveconventionally been proposed, and thus can be used as appropriate. Theproposed methods include normalized correlation calculation such asZero-mean Normalized Cross-Correlation (ZNCC), and Sum of AbsoluteDifference (SAD) based on the sum of absolute differences between thewaveforms.

The images do not necessarily need to be converted into monochromeimages. The phase difference may be detected by using red components R,green components G, blue components B, and infrared components in theleft pupil image IL(x) and the superimposed image ILR′(x). When theobject has unbalanced color components, the color componentcorresponding to the highest imaging sensitivity and the highest SNratio can be effectively used for the detection.

When the image sensor 40 has sensitivity covering the near infraredwavelength band, the object 5 may be selectively irradiated with visiblelight or infrared light. Then, a visible image or a near infrared imagecan be selectively obtained as the observation image IL(x) and thesuperimposed image ILR′(x) for measurement obtained with two pupils. Thevisible image may be obtained to pursue color purity. The visible imageand the near infrared image may be simultaneously obtained to pursue thehigh sensitivity and the high SN ratio. Only the near infrared image maybe obtained for special purposes.

The phase difference detection section 330 outputs the phase differenceinformation thus detected, and the error flag to the distanceinformation calculation section 360. The distance informationcalculation section 360 calculates the distance information (forexample, the distance z in FIG. 8) on the object 5 for each pixel, andoutputs the resultant distance information to the three-dimensionalinformation generating section 370. For example, the pixel provided withthe error flag may be regarded as a flat portion of the object 5 (anarea with a small amount of edge components), and interpolation may beperformed for such pixel based on the distance information onsurrounding pixels. The three-dimensional information generating section370 generates three-dimensional information from the distanceinformation (or from the distance information and the RGB image from thecolor image generating section 320). The three-dimensional informationmay be various types of information including a Z value map (distancemap), polygon, and a simulative-three-dimensional display image (withshape emphasized by shading or the like, for example). Thethree-dimensional information generating section 370 generates athree-dimensional image and three-dimensional data generated, or adisplay image obtained by superimposing the observation image on theimage as appropriate, and outputs the resultant image and/or data to thedisplay section 220. The display section 220 displays thethree-dimensional information.

The movable mask position detection section 350 detects whether themovable mask 30 is at the position corresponding to the observation modeor at the position corresponding to the stereoscopic measurement mode byusing the images IL(x) and ILR′(x) used in the stereoscopic measurementmode. When the movable mask 30 is in the state not matching the mode, aposition error flag is output to the movable mask control section 340.Upon receiving the position error flag, the movable mask control section340 corrects the movable mask 30 to be in the correct state (statecorresponding to the image selection). When the correction operationcannot achieve the correct state, some sort of failure is determined tohave occurred, and thus the function of the entire system is stopped.

For example, whether the movable mask 30 is at the positioncorresponding to the observation mode or is at the positioncorresponding to the stereoscopic measurement mode determined throughthe following methods 1 to 4. One or a plurality of the first to thefourth methods may be employed.

In the first method, whether or not an average value of the phasedifferences s within a predetermined region of the image is of anegative value is determined. This method is for a case where themovable mask 30 erroneously closes the left-eye optical path under theobservation mode. In such a case, the right pupil image IR(x) isobtained as the reference image, which is supposed to be the left pupilimage IL(x), leading to reversed positional relationship between theimages IL′(x) and IR′(x) forming the superimposed image ILR′(x),resulting in the phase difference s of a negative value.

In the second method, whether or not the matching evaluation value fordetecting the phase difference s is equal to or lower than apredetermined value is determined. This method is for a case where theleft-or the right-eye optical path is incompletely closed under theobservation mode. In such a case, matching is evaluated between doubleimages with different profiles. Such evaluation results in a lowmatching evaluation value obtained with the phase difference s that issupposed to indicate the match.

In the third method, whether or not the average value of the phasedifferences s within a predetermined region of the image is equal to orsmaller than a predetermined value (a value close to 0) is determined.This method is for a case where the left- or the right-eye optical pathis closed under the stereoscopic measurement mode. In such a case, animage with no phase difference s over the entire imaging area isobtained. Thus, the phase difference s is substantially 0.

In the fourth method, whether or not a brightness ratio between theobservation image IL(x) and the superimposed image ILR′(x) formeasurement is within a predetermined range is determined. When themovable mask 30 is properly operating, a substantially constantbrightness ratio between the images is achieved.

8. Mode Switching Sequence

FIG. 10 illustrates a first sequence (first timing chart) of a moviecapturing operation.

As illustrated in FIG. 10, switching of the state of the movable mask30, an image capturing timing, and selection of the captured image areinterlocked. As indicated by A1 and A2, the mask state corresponding tothe observation mode and the mask state corresponding to thestereoscopic measurement mode are alternately achieved. As indicated byA3 and A4, an image is captured each time the mask state changes. Asindicated by A5, the image captured with the image sensor 40 in a framefn with the mask state corresponding to the observation mode is selectedas an observation image IL(x). As indicated by A6, the image capturedwith the image sensor 40 in a frame fn+1 with the mask statecorresponding to the stereoscopic measurement mode is selected as themeasurement image ILR′(x).

With the observation mode and the stereoscopic measurement mode thusalternately repeated, the observation image IL(x) and the measurementimage ILR′(x) can be contiguously obtained substantially in real time.Thus, the monitoring and the measurement can both be achieved even whenthe object 5 moves. When the observation image IL(x) is displayed withmeasurement information overlaid as appropriate, useful information canbe provided so that the user can perform visual inspection andquantitative inspection at the same time.

The measurement processing is executed by using the observation imageIL(x) and the measurement image ILR′(x) subsequently obtained, or byusing the measurement image ILR′(x) and the observation image IL(x)subsequently obtained. For example, as indicated by A7, measurementprocessing Mn+1 is executed with the observation image IL(x) captured inthe frame fn and the measurement image ILR′(x) captured in the framefn+1, in a measurement period after an image capturing period in theframe fn+1. Alternatively, as indicated by A8, the observation imageIL(x) is captured in a frame fn+2, and as indicated by A9, measurementprocessing Mn+2 is executed with the measurement image ILR′(x) capturedin the frame fn+1 and the observation image IL(x) captured in fn+2, in ameasurement period after the image capturing period in the frame fn+2.Thus, measurement information can be obtained substantially in real timein each frame.

FIG. 11 illustrates a second sequence (second timing chart) ofoperations in a movie capturing.

In FIG. 11, a mask state for the observation mode is set in a singleframe as indicated by B1, and a mask state for the stereoscopicmeasurement mode is set in a plurality of subsequent frames as indicatedby B2. FIG. 11 illustrates an example where the plurality of frames arefive frames. However, this should not be construed in a limiting sense.A single image is captured with the mask state for the observation modeas indicated by B3, and the image thus captured in the frame fn isselected as the observation image IL(x) as indicated by B4. Five imagesare captured with the mask state for the stereoscopic measurement modeas indicated by B5, and these images captured in frames fn+1 to fn+5 areeach selected as the measurement image ILR′(x) as indicated by B6.

The frames fn+1 to fn+5 are referred to as fn+i (i is an integersatisfying 1≤i≤5). In the measurement period after the image capturingperiod in the frame fn+i, the measurement processing Mn+i is executedwith the observation image IL(x) captured in the frame fn and themeasurement image ILR′(x) captured in the frame fn+i. Then, the phasedifference s′(xL) is obtained through the following Formula (11). In theformula, s′(xL)i represents a phase difference at a certain pixel(coordinate) xL on the image sensor 40 in the frame fn+i, j is thenumber of measurement images ILR′(x) captured for the single observationimage IL(x). In FIG. 11, j is five.

$\begin{matrix}{{s^{\prime}({xL})} = {\frac{1}{j}{\sum\limits_{i = 1}^{j}{{s^{\prime}({xL})}i}}}} & (11)\end{matrix}$

With a phase differences s′(xL)n thus obtained integrated and averagedwithin the frames (fn+1 to fn+5), a more accurate phase differences′(xL) with can be obtained with small fluctuation. The phase differences′(xL)n is preferably obtained with the influence of the movementbetween frames eliminated (using the method illustrated in FIG. 7).

9. Second Configuration Example

FIG. 12 and FIG. 13 illustrate a second basic configuration example ofan imaging section of an endoscope apparatus. FIG. 14 and FIG. 15illustrate a second detailed configuration example of the fixed mask 20and the movable mask 30. FIG. 12 and FIG. 13 are each a cross-sectionalside view of the imaging section (as viewed along a plane including anoptical axis) and illustrate relationship between an amount of light ofan image formed on the image sensor (or a pixel value of the imagecaptured with the image sensor) and the position x. FIG. 14 and FIG. 15each include a cross-sectional view of the imaging optical system 10,the fixed mask 20, and the movable mask 30, and a diagram illustratingthe fixed mask 20 and the movable mask 30 as viewed in the optical axisdirection (a back view as viewed from the image side). The componentsthat are the same as those described above with reference to FIG. 1 toFIG. 4 are denoted with the same reference signs, and the descriptionthereof is omitted as appropriate.

As illustrated in FIG. 12 to FIG. 15, the imaging optical system 10 mayinclude a monocular optical system. The monocular optical systemincludes one or a plurality of lenses. The monocular optical system hasa single pupil divided into a left pupil and a right pupil with the stopholes 21 and 22 of the fixed mask 20. Center lines IC1 and IC2 aredefined as lines that pass through the centers of the stop holes 21 and22 (the center of a circle when the stop hole has a circular shape forexample) and are in parallel with an optical axis AXC of the imagingoptical system 10. For example, the center lines IC1 and IC2 arearranged to be at an equal distance from the optical axis AXC. Forexample, the fixed mask 20 is provided at a pupil position of theimaging optical system 10. The movable mask 30 is set to be at aposition to shield the stop hole 22 from light in the observation modeillustrated in FIG. 12 and FIG. 14. The movable mask 30 is set to be ata position to open the stop holes 21 and 22 in the stereoscopicmeasurement mode illustrated in FIG. 13 and FIG. 15.

In this configuration example, the area φL of the stop hole 21 isdifferent from the area φR of the stop hole 22. For example, the stophole 22 is smaller than the stop hole 21. In FIG. 12 to FIG. 15, φL>φRholds true. However, this should not be construed in a limiting sense,and a configuration satisfying φL<φR may be employed.

In the monocular optical system, the phase difference s is 0 at aposition where the object is in focus, and becomes a positive ornegative value (the right pupil image IR′ is shifted from the left pupilimage IL′ on one of opposite sides) with the focal point shifted forwardor rearward. In the present embodiment, the left pupil image IL′ and theright pupil image IR′ have different brightness values, so that the sideon which the right pupil image IR′ is shifted from the left pupil imageIL′ can be determined from the superimposed image.

In the stereoscopic optical system illustrated in FIG. 1 to FIG. 4, thephase difference s remains to be a positive or negative value (the rightpupil image IR′ is shifted from the left pupil image IL′ toward the sameside) regardless of the focus status. Thus, the left pupil image IL′ andthe right pupil image IR′ do not need to have different values ofbrightness.

This configuration example employs a method that is the same as thatdescribed above with reference to FIG. 6 or FIG. 7 to detect a phasedifference s(x) from the image IL(x) captured in the observation modeand the image ILR′(x) captured in the stereoscopic measurement mode.Similarly, the endoscope apparatus according to this configurationexample may have a configuration similar to that illustrated in FIG. 5and FIG. 9.

10. Third Configuration Example

FIG. 16 and FIG. 17 illustrate a third detailed configuration example ofthe fixed mask 20 and the movable mask 30. FIG. 16 and FIG. 17 eachinclude a cross-sectional view of the imaging optical system 10, thefixed mask 20, and the movable mask 30, and a diagram illustrating thefixed mask 20 and the movable mask 30 as viewed in the optical axisdirection (a back view as viewed from the image side). The componentsthat are the same as those described above with reference to FIG. 1 toFIG. 4 and FIG. 12 to FIG. 15 are denoted with the same reference signs,and the description thereof is omitted as appropriate.

As illustrated in FIG. 16 and FIG. 17, the imaging optical system 10 mayinclude a single optical system. The fixed mask 20 has the lightshielding section 24 provided with a single stop hole 23 (through hole).In the observation mode illustrated in FIG. 16, the movable mask 30 isset to be at the position for opening the stop hole 23. In thestereoscopic measurement mode illustrated in FIG. 17, the movable mask30 is set to be at a position for dividing the stop hole 23 into twoholes (referred to as FL and FR). With these holes FL and FR, a singlepupil of the monocular optical system is divided into the left pupil andthe right pupil.

For example, the stop hole 23 has a circular shape and has the centerline (the center of the circle) matching the optical axis AXC. Themovable mask 30 has the light shielding section with a width (width in adirection orthogonal to the longitudinal direction) smaller than thesize (diameter) of the stop hole 23. In the stereoscopic measurementmode, the movable mask 30 is configured in such a manner that the holesFL and FR have different areas φL and φR. For example, the rotationalshaft 35 may be arranged in an eccentric manner so that the center lineof the light shielding section of the movable mask 30 in thelongitudinal direction does not pass through the center of the stop hole23 when the rotational angle is 0.

With this configuration example, the stop hole 23 with a large openingcan be used, whereby the bright observation image IL(x) can be captured.The stop hole 23 has the center line matching the optical axis AXC sothat a high quality image (with features such as small distortion and alarge angle of view) can be captured by using the light passing throughthe center of the optical axis of the imaging optical system 10.

This configuration example employs a method that is the same as thatdescribed above with reference to FIG. 6 or FIG. 7 to detect the phasedifference s(x) from the image IL(x) captured in the observation modeand the image ILR′(x) captured in the stereoscopic measurement mode.Similarly, the endoscope apparatus according to this configurationexample may have a configuration similar to that illustrated in FIG. 5and FIG. 9.

11. Principle of Stereoscopic Three-Dimensional Measurement

The principle of the stereoscopic three-dimensional measurement in thesecond configuration example illustrated in FIG. 12 to FIG. 15 and thethird configuration example illustrated FIG. 16 and FIG. 17 isdescribed.

As illustrated in FIG. 13, an X axis and a Y axis orthogonal to the Xaxis are set along the image sensor plane. A Z axis, toward the object,is set to be in a direction that is orthogonal to the image sensorplane, and parallel to the optical axis AXC. The Z axis, the X axis, andthe Y axis intersect at the zero point. The Y axis is omitted for thesake of illustration.

An appropriate distance between the imaging optical system 10 (imaginglens) and a measurement point of the object is defined as z, and adistance between an end of z and the focal point is defined as b′. Adistance between the imaging optical system 10 and a referencemeasurement point is defined as a, and a distance between the imagingoptical system 10 and an image sensor plane is defined as b. Thereference measurement point is a point with which the focal point on theimage sensor plane is achieved. The left and the right pupils (thecenters of gravities of the pupils) are separated from each other by adistance d. An X coordinate of the center of gravity of an image of acertain point P(x,y) of the object formed with the left pupil on theimage sensor plane is defined as xL, and an X coordinate of the centerof gravity of an image of the point P(x,y) of the object formed with theright pupil on the image sensor plane is defined as xR. The followingformulae can be obtained based on similar relationship among a pluralityof triangular sections formed in a triangle defined by lines connectinga certain point P(x,z), a focal point P′(x,z), and the coordinatesxL,xR.

The following Formula (12) represents the phase difference scorresponding to the shifting amount between the left pupil image andthe right pupil image. The value s is a positive value, a negativevalue, or 0.s=xR−xL  (12)

The following Formula (13) can be obtained based on the similarrelationship between triangles.

$\begin{matrix}{\frac{s}{d} = \frac{b - b^{\prime}}{b^{\prime}}} & (13)\end{matrix}$

The following Formulae (14) and (15) are obtained based on the principleof focus relationship, with f representing the focal distance of theimaging optical system 10.

$\begin{matrix}{{\frac{1}{a} + \frac{1}{b}} = \frac{1}{f}} & (14) \\{{\frac{1}{z} + \frac{1}{b^{\prime}}} = \frac{1}{f}} & (15)\end{matrix}$

The following Formula (16) can be obtained by removing b′ and f fromFormulae (13) to (15).

$\begin{matrix}{z = \frac{({ab}) \cdot d}{{b \cdot d} - {a \cdot s}}} & (16)\end{matrix}$

Because a, b, and d are known setting values, the distance z to theobject can be obtained if the phase difference s can be obtained. Theshape of the object can be measured by detecting the position xRcorresponding a position xL on the image sensor plane based on the phasedifference s by matching processing (correlation calculation), andcalculating the distance z for all of the positions xL. Note that thedistance z may not be obtainable at a position where favorable matchingis unobtainable.

12. Reason why Stop Holes have Different Areas

In the second configuration example illustrated in FIG. 12 to FIG. 15and in the third configuration example illustrated in FIG. 16 and FIG.17, the opening areas φL and φR of the stop holes 21 and 22 (or FL andFR) of the left and the right pupil paths are intentionally set to bedifferent from each other.

The reason why such a configuration is employed is described below usingthe distances z, b′, a, and b, and the positions xL and xR illustratedin FIG. 13. The direction in which the right pupil image is shifted fromthe left pupil image on the image sensor plane is opposite between caseswhere the distance between the imaging optical system 10 (imaging lens)and the measurement point is shorter than the reference distance a andin a case where the distance is longer than the reference distance a.Specifically, the relationship expressed in the following Formula (17)is satisfied, so that a larger one of z and a changes from one to theother based on the shifted direction.

$\begin{matrix}\left. \begin{matrix}{{xL} < {{{xR}\left( {{{xR} - {xL}} > 0}\; \right)}\mspace{20mu}{when}\mspace{14mu} z} < a} \\{{xL} = {{{xR}\mspace{14mu}{when}\mspace{14mu} z} = a}} \\{{xL} > {{{xR}\left( {{{xR} - {xL}} < 0}\; \right)}\mspace{20mu}{when}\mspace{14mu} z} > a}\end{matrix} \right\} & (17)\end{matrix}$

Thus, when the left pupil image IL′(x) and the right pupil image ’IR(x)can be regarded as substantially the same, an image obtained bysuperimposing these images with the same brightness does not enable theobserver to recognize which one of the superimposed images has shiftedin which one of left and right directions.

In view of this, in the present embodiment, the areas φL and φR of thestop holes 21 and 22 (or FL and FR) of the left and the right pupilpaths are intentionally set to be different from each other. If the leftpupil image IL′(x) and the right pupil image IR′(x) have differentvalues of brightness, the superimposed image ILR′(x) varies depending onwhich one of the left and right direction the right pupil image IR′(x)has shifted relative to the left pupil image IL′(x). This enables theobserver to recognize the shifted direction.

If the left pupil image IL′(x) and the right pupil image IR′(x) havedifferent values of brightness, the superimposed image ILR′(x) is likelyto be more different among superimposing manners, whereby the shiftingamount can be more accurately detected.

13. Third Method for Detecting Phase Difference

In FIG. 6 and FIG. 7, the phase difference s is detected under anassumption that the area ratio (φR/φL) between the stop holes 21 and 22is the same over the entire pixel positions. However, the componentratio between IL′(x) and IR′(x) in the superimposed image ILR′(x) mightvary among pixel positions. In such a case, the area ratio (φR/φL)cannot be regarded as being the same. For example, the profile of theimage ILR′(x) varies due to the change in the viewpoint relative to theobject and the imaging position on the image sensor plane. In such acase, the accuracy of matching using the combined image ILR(x,δ,s) mightbe compromised in actual use.

In view of this, the present method includes calculating the componentratio between the image IL′(x) and the image IR′(x) each time the ratiochanges, and generating the combined image with the image IL(x) and theimage IR(x) with the component ratio matching that between the imageIL′(x) and the image IR′(x). Thus, the matching evaluation can beaccurately executed.

As illustrated in FIG. 18, vectors VL, VR, RL, and RR are eachillustrated as a sampling sequence (a pixel value sequence) within apredetermined with w.

The vector VL is a sampling sequence in the image IL(x,δ). The vector VRis a sampling sequence in the image IL(x,δ,s). A composite vector CV isa sampling sequence in the image ILR(x,δ,s). These images are generatedfrom images captured in the observation mode, and are defined by Formula(3) and (4) described above. The vector RL is a sampling sequence in theimage IL′(x). The vector RR is a sampling sequence in the image IR′(x).A superimposed vector CR is a sampling sequence in the image ILR′(x).The image ILR′(x) is an image captured in the stereoscopic measurementmode and is formed by the images IL′(x) and IR′(x).

Coordinates xk, xk′ of each vector component are defined as in thefollowing Formula (18) based on a certain sampling position xL. In thefigure, K represents the number of sampling times in the section w (thenumber of pixels in the section w in the parallax direction (xdirection)).xk=xL+δ−(w/2)+k,xk′=xL+δ′−(w/2)+k(k=0,1,2, . . . ,K)  (18)

Using this Formula (18), the components of each vector can be expressedas in the following Formulae (19) and (20).

$\begin{matrix}\left. \begin{matrix}{{VL} = \left\lbrack {{{IL}\left( {{x\; 0} - \delta} \right)},{{IL}\left( {{x\; 1} - \delta} \right)},{{IL}\left( {{x\; 2} - \delta} \right)},\ldots\mspace{14mu},{{IL}\left( {{xK} - \delta} \right)}} \right\rbrack} \\{{VR} = \left\lbrack {{{IL}\left( {{x\; 0} - \delta - s} \right)},{{IL}\left( {{x\; 1} - \delta - s} \right)},{{IL}\left( {{x\; 2} - \delta - s} \right)},\ldots\mspace{14mu},{{IL}\left( {{xK} - \delta - s} \right)}} \right\rbrack} \\{{CV} = {{VL} + {VR}}}\end{matrix} \right\} & (19) \\\left. \begin{matrix}{{RL} = \left\lbrack {{{IL}^{\prime}\left( {{x\; 0^{\prime}} - \delta^{\prime}} \right)},{{IL}^{\prime}\left( {{x\; 1^{\prime}} - \delta^{\prime}} \right)},{{IL}^{\prime}\left( {{x\; 2^{\prime}} - \delta^{\prime}} \right)},\ldots\mspace{14mu},{{IL}^{\prime}\left( {{xK}^{\prime} - \delta^{\prime}} \right)}} \right\rbrack} \\{{RR} = \left\lbrack {{{IL}^{\prime}\left( {{x\; 0^{\prime}} - \delta^{\prime} - s^{\prime}} \right)},{{IL}^{\prime}\left( {{x\; 1^{\prime}} - \delta^{\prime} - s^{\prime}} \right)},{{IL}^{\prime}\left( {{x\; 2^{\prime}} - \delta^{\prime} - s^{\prime}} \right)},\ldots\mspace{14mu},{{IL}^{\prime}\left( {{xK}^{\prime} - \delta^{\prime} - s^{\prime}} \right)}} \right\rbrack} \\{{CR} = {{RL} + {RR}}}\end{matrix} \right\} & (20)\end{matrix}$

The image IL(x,δ) and the image IL′(x) can be regarded as being insimilar relationship, and the image IL(x,δ,s) and the image IR′(x) canbe regarded as being in similar relationship. The following Formula (21)can be obtained with gL representing a correction coefficient for themagnitude of a vector in a case where the image IL(x,δ) and the imageIL′(x) match, and gR representing a correction coefficient for themagnitude of a vector in a case where the image IL(x,δ,s) and the imageIR′(x) match.RL=gL·VL,RR=gR·VR  (21)

With the correction coefficients gL and gR in Formula (21) describedabove obtained, the composite vector CV and the superimposed vector CRmatch if the positional relationship matches between the image IL(x,δ)and the image IL′(x) and the positional relationship matches between theimage IL(x,δ,s) and the image IR′(x).

The motion amount δ and the phase difference s can be searched for bydetecting the position at which the vector CV and the vector CR matchwhile correcting the magnitudes of the vectors VL and VR in Formula (21)described above. Thus, the motion amount δ and the phase difference scan be accurately obtained. In other words, the vectors VL and VR, whichare components of the vector CV, are normalized to match the componentratio between the vectors RL and RR, and then the vector CV and thevector CR are compared with each other. Thus, the matching evaluationcan be properly performed with the matching level between the vector CRand the vector CV increased.

First of all, the magnitudes of the vectors RL and RR, which arecomponents of the detected vector CR corresponding to the superimposedimage ILR′(x), are obtained. The image IL(x,δ) and the image IL′(x) aswell as the image IL(x,δ,s) and the image IR′(x) are regarded as beingin similar relationship within a limited calculation range w. Thus,directions of the vector VL and the vector RL as well as directions ofthe vector VR and the vector RR can be assumed to match if the positionsof the vectors match even when the vectors have difference magnitudes.FIG. 19 is a schematic view illustrating relationship among vectors.

An angle between the vector CR and the vector RL, that is, an anglebetween the vector CR and the vector VL is defined as a. An anglebetween the vector CR and the vector RR, that is, an angle between thevector CR and the vector VR is defined as β. The relationship in thefollowing Formula (22) can be obtained with the angles α and β.

$\begin{matrix}\left. \begin{matrix}{{{VL} \cdot {CR}} = {{{VL}}{{CR}}\cos\;\alpha}} \\{{{VR} \cdot {CR}} = {{{VR}}{{CR}}\cos\;\beta}} \\{\gamma = {{\pi/2} - \left( {{\cos\;\alpha} + {\cos\;\beta}} \right)}}\end{matrix} \right\} & (22)\end{matrix}$

Angles α, β, and γ are obtained with Formula (22) described above andare substituted into the following Formula (23). Thus, magnitudes |RL|and |RR| of the vectors RL and RR can be obtained.

$\begin{matrix}\left. \begin{matrix}{{{RL}} = {{{{CR}}\cos\;\alpha} - {{{CR}}\sin\;{\beta \cdot \tan}\;\gamma}}} \\{{{RR}} = {{{{CR}}\cos\;\beta} - {{{CR}}\sin\;{\alpha \cdot \tan}\;\gamma}}}\end{matrix} \right\} & (23)\end{matrix}$

The following Formula (24) is obtained based on Formula (21) describedabove.|RL|=gL·|VL|,|RR|=gR·|VR|  (24)

The following Formula (25) is obtained based on Formula (24) describedabove.

$\begin{matrix}{{{gL} = \frac{{RL}}{{VL}}},{{gR} = \frac{{RR}}{{VR}}}} & (25)\end{matrix}$

The correction coefficient gL and gR can be obtained by substituting|RL| and |RR|, obtained with Formula (23) described above, in to Formula(25) described above.

A vector NCV is newly obtained by combining the vector VL and VR afterhaving the component amounts (magnitudes) corrected by using thecorrection coefficients gR and gL thus obtained. The following Formula(26) is obtained based on Formula (21) described above.NCV=gL·VL+gR·VR=RL+RR  (26)

All things considered, when the vector CR and the vector NCV match, thepositional relationship substantially match between the vector VL andthe vector RL and between the vector VR and the vector RR.

Note that the directions of the vector RL and the vector RR are the sameif the superimposed image ILR′(x) involves no phase difference s′ as inthe following Formula (27). In such a case the coefficients gL and gRcannot be identified from the calculation method described above.α=0 and β=0  (27)

Still, in such a case, the direction of the search vector VL matches thedirection of the vector RL, and the direction of the search vector VRmatches the vector RR. Note that the directions might be closest as muchas possible instead of matching due to a degrading factor such as noise.

Thus, a position at which the angle α and the angle β are both thesmallest as much as possible is identified and evaluated as the positionwhere the motion amount δ′ matches. The angle α and the angle β are bothdetermined to be the smallest as much as possible with a smallest valueof (α+β) (a value close to 0), where α>0 and β>0. For example, amatching evaluation function E as in the following Formula (28) is used.E=(α+β)·|NCV−CR|  (28)

The evaluation function is not limited to this, and may beE=(α+β)·(1−NCC[NCV,CR]) for example. NCC[NCV,CR] is a value indicatingcorrelation between vectors NCV and CR, obtained with ZNCC.

The correction coefficients gL and gR of appropriate values are obtainedwith Formula (21) to (25) described above only when the vector CV andthe vector CR match. In other words, only the correction coefficients gLand gR of inappropriate values (invalid values) are obtained during thesearch for the phase difference s and the motion amount δ, if the vectorCV and the vector CR do not match. Thus, a case of applying thecorrection coefficients gL and gR obtained with Formulae (21) to (25)described above without successful matching, that is, with thedirections of the vectors CV and CR not matching is more likely toresult in a failure to match the vectors CV and CR than in a case of notapplying the coefficients. This feature is favorable in terms ofmatching position detection.

The embodiments and the modifications thereof according to the presentinvention are described. However, the present invention is not limitedthe embodiments and the modifications only, and the present inventioncan be implemented with the elements modified without departing from thegist of the invention. The plurality of elements disclosed in theembodiments and the modifications may be combined as appropriate toimplement the invention in various ways. For example, some of all theelements described in the embodiments and the modifications may bedeleted. Furthermore, elements in different embodiments andmodifications may be combined as appropriate. Thus, various modificationand application can be made without departing from the gist of thepresent invention. Any term cited with a different term having a broadermeaning or the same meaning at least once in the specification and thedrawings can be replaced by the different term in any place in thespecification and the drawings.

What is claimed is:
 1. An imaging device comprising: an image sensor; anoptical system forming an image of an object on the image sensor; and aprocessor, the optical system switching between a first state ofcapturing an image of the object with a single pupil and a second stateof capturing an image of the object with two pupils, the processor beingconfigured to implement generating a simulative phase difference imagefrom a first captured image captured with the image sensor in the firststate, and executing matching processing of comparing the simulativephase difference image with a second capture image captured with theimage sensor in the second state to detect a phase difference between animage formed with one of the two pupils and an image formed with anotherone of the two pupils.
 2. The processor as defined in claim 1, theprocessor being configured to implement generating a first simulativepupil image corresponding to the image formed with the one of the pupilsand a second simulative pupil image corresponding to the image formedwith the other one of the pupils, from the first captured image,generating the simulative phase difference image through processing ofadding together the first simulative pupil image and the secondsimulative pupil image shifted from each other by a shifting amountcorresponding to the phase difference, and detecting the phasedifference through the matching processing while changing the shiftingamount.
 3. The imaging device as defined in claim 1, the two pupils ofthe optical system have difference sizes.
 4. The imaging device asdefined in claim 3, the processor being configured to implementgenerating the first simulative pupil image corresponding to the imageformed with the one of the pupils and the second simulative pupil imagecorresponding to the image formed with the other one of the pupils, withgain adjustment, based on the different sizes of the two pupils,executed on the first captured image, generating the simulative phasedifference image through processing of adding together the firstsimulative pupil image and the second simulative pupil image shiftedfrom each other by a shifting amount corresponding to the phasedifference, and detecting the phase difference through the matchingprocessing while changing the shifting amount.
 5. The imaging device asdefined in claim 1, the processor being configured to implement furtherdetecting a motion amount due to an object moving between the firstcaptured image and the second captured image, based on the firstcaptured image and the second captured image.
 6. The imaging device asdefined in claim 5, the processor being configured to implementgenerating a first simulative pupil image corresponding to the imageformed with the one of the pupils and a second simulative pupil imagecorresponding to the image formed with the other one of the pupils, fromthe first captured image, generating the simulative phase differenceimage through processing of adding together the first simulative pupilimage and the second simulative pupil image shifted from each other by afirst shifting amount corresponding to the phase difference and by asecond shifting amount corresponding to the motion amount, and detectingthe phase difference and the motion amount through the matchingprocessing while changing the first shifting amount and the secondshifting amount independently from each other.
 7. The imaging device asdefined in claim 1, the optical system is set to be in the first statein an n-th frame and is set to be in the second state in n+1-th ton+j-th frames after the n-th frame, n being an integer, j being aninteger that is equal to or larger than two, the processor beingconfigured to implement detecting the phase difference, based on thefirst captured image captured in the n-th frame and the second capturedimage captured in an n+i-th frame, in the n+1-th to the n+j-th frames,for j times, and executing processing of averaging the j phasedifferences, i being an integer that is equal to larger than one and isequal to or smaller than j.
 8. The imaging device as defined in claim 1,the processor being configured to implement outputting an observationimage based on the first captured image.
 9. The imaging device asdefined in claim 1, the optical system including: a fixed mask includinga first opening and a second opening; and a movable mask that is movablerelative to the fixed mask, in the first state, the movable mask notclosing the first opening and closing the second opening, and theoptical system forming the image of the object using the first openingas the single pupil, in the second state, the movable mask not closingthe first opening or the second opening, and the optical system formingthe image of the object using the first opening and the second openingas the two pupils.
 10. The imaging device as defined in claim 9, thesecond opening being smaller than the first opening in the fixed mask.11. The imaging device as defined in claim 1, the optical systemincluding: a fixed mask including an opening; and a movable mask that ismovable relative to the fixed mask, in the first state, the movable masknot splitting the opening, and the optical system forming the image ofthe object using the opening not split as the single pupil, in thesecond state, the movable mask splitting the opening into a first splitopening and a second split opening smaller than the first split opening,and the optical system forming the image of the object using the firstsplit opening and the second split opening as the two pupils.
 12. Theimaging device as defined in claim 1, the processor being configured toimplement performing control to switch between a first mode of settingthe optical system to be in the first state and a second mode of settingthe optical system to be in the second state.
 13. The imaging device asdefined in claim 12, the processor being configured to implementdetecting at least one of the optical system set to be in the firststate under the first mode and the optical system set to be in thesecond state under the second mode, based on an image captured under thefirst mode and an image captured under the second mode.
 14. An endoscopeapparatus comprising the imaging device as defined in claim
 1. 15. Animaging method comprising: switching a state of an optical systembetween a first state in which the optical system forms an image of anobject on an image sensor with one pupil and a second state in which theoptical system forms the image of the object on the image sensor withtwo pupils, generating a simulative phase difference image from a firstcaptured image captured with the image sensor in the first state,executing matching processing to compare the simulative phase differenceimage with a second captured image captured with the image sensor in thesecond state, and detecting a phase difference between an image formedwith one of the two pupils and an image formed with another one of thetwo pupils.